17/03/2024 - 23/03/2024

17/03/2024 22:07

I found the reason and "solution" for this change:

Change the magic number in this function to correspond to our IP. I hate this code the more I read it.
MasterGM2/frontend.cpp:595
+strcpy(master_settings_odb.encoder_fe, "AMC13000");
MasterGM2/frontend.cpp:595
+strcpy(master_settings_odb.encoder_fe, "AMC13000");
overwrite this setting that was set to "AMC1300" as opposed to "AMC13000" I'm not sure where it originally gets set. I think the extra 0 is a result of Lawrence's hijinx to make frontends have 3 digits as opposed to 2.

First off, the reason this change needs to be made is because the change to midas to allow frontends to have 3 digit labels.

mfe.cxx line 2580: sprintf(full_frontend_name + strlen(full_frontend_name), "%02d", frontend_index); --> >sprintf(full_frontend_name + strlen(full_frontend_name), "%03d", frontend_index);

But, the master_Settings_odb.encoder_fe is actually the ODB setting under Equipment/MasterGM2/Settings/Globals/Encoder Front End. To make everything "work" as Lawrence intended I had to do a few things:

Setup environment

cd /home/installation_testing/packages/experiment/lxedaq/environment_setup
source ./setup_environment.sh

cd /home/installation_testing/packages/experiment/lxedaq/environment_setup
source ./setup_environment.sh

Nuke the ODB

$MIDASSYS/bin/odbinit -s 1024MB --cleanup

$MIDASSYS/bin/odbinit -s 1024MB --cleanup

Initialize ODB
First let's generate the MasterGM2 settings:

cd /home/installation_testing/packages/experiment/lxedaq/frontends/CaloReadoutAMC13
./frontend -e DAQ

cd /home/installation_testing/packages/experiment/lxedaq/frontends/CaloReadoutAMC13
./frontend -e DAQ

It should crash, but generate the ODB path Equipment/MasterGM2/Settings/Globals. Under this path, change Front End Offset to 1 and Encoder Front End to AMC13001. Now we move on to initializing the AMC13001 frontend.
Make sure /home/installation_testing/packages/experiment/lxedaq/frontends/AMC13xx_config.xml looks something like below (notice frontend id="0")

<?xml version="1.0" encoding="UTF-8"?>
<frontend id="0">
    <slot id="5"  type="WFD" />
    <slot id="11" type="FC7" />
</frontend>

<?xml version="1.0" encoding="UTF-8"?>
<frontend id="0">
    <slot id="5"  type="WFD" />
    <slot id="11" type="FC7" />
</frontend>

Then run

cd /home/installation_testing/packages/experiment/lxedaq/frontends/
./initialize_ODB.sh

cd /home/installation_testing/packages/experiment/lxedaq/frontends/
./initialize_ODB.sh

Rename Equipment/AMC13000 --> Equipment/AMC13001 in the ODB, the run:

cd /home/installation_testing/packages/experiment/lxedaq/frontends/CaloReadoutAMC13
./frontend -i 1 -e DAQ

cd /home/installation_testing/packages/experiment/lxedaq/frontends/CaloReadoutAMC13
./frontend -i 1 -e DAQ

this will fill AMC13001 with the proper Settings, Common, Variables, Statistics, and Monitors information. Finally, change
/home/installation_testing/packages/experiment/lxedaq/frontends/AMC13xx_config.xml frontend id to look something like below

<?xml version="1.0" encoding="UTF-8"?>
<frontend id="1">
    <slot id="5"  type="WFD" />
    <slot id="11" type="FC7" />
</frontend>

<?xml version="1.0" encoding="UTF-8"?>
<frontend id="1">
    <slot id="5"  type="WFD" />
    <slot id="11" type="FC7" />
</frontend>

4.Now we can run the master frontend without hardcoding the frontend number

cd /home/installation_testing/packages/experiment/lxedaq/frontends/CaloReadoutAMC13
./frontend -e DAQ

cd /home/installation_testing/packages/experiment/lxedaq/frontends/CaloReadoutAMC13
./frontend -e DAQ

Currently, I'm still stuck with the SFP port error:

FC7 Initialization Started
FC7_init(174): FC7 Board Presence Check
FC7_init(190): Slot 11: Read FC7 IP Address: 192.168.1.11
FC7_init(241): FC7 Ethernet Communication Check: 1/1
FC7_init(288): Slot 11: FC7 Firmware Hard Reset
FC7_init(298): Waiting 5 s ...
FC7_init(410): Slot 11: Write: Enabled Top SFP Ports
top: 1, enable: 1
FC7_init(420): Waiting 15 s ...
top_mask[10]: 1
FC7_init(450): Slot 11: Read: Enabled Top SFP Ports: 0
[MasterGM2,ERROR] [AMC1300.cpp:451:FC7_init,ERROR] /AMC1300/Settings/FC7-11/: Enabled Top SFP Ports Failure
[MasterGM2,ERROR] [frontend.cpp:904:frontend_init,ERROR] FC7 Initialization Failed

FC7 Initialization Started
FC7_init(174): FC7 Board Presence Check
FC7_init(190): Slot 11: Read FC7 IP Address: 192.168.1.11
FC7_init(241): FC7 Ethernet Communication Check: 1/1
FC7_init(288): Slot 11: FC7 Firmware Hard Reset
FC7_init(298): Waiting 5 s ...
FC7_init(410): Slot 11: Write: Enabled Top SFP Ports
top: 1, enable: 1
FC7_init(420): Waiting 15 s ...
top_mask[10]: 1
FC7_init(450): Slot 11: Read: Enabled Top SFP Ports: 0
[MasterGM2,ERROR] [AMC1300.cpp:451:FC7_init,ERROR] /AMC1300/Settings/FC7-11/: Enabled Top SFP Ports Failure
[MasterGM2,ERROR] [frontend.cpp:904:frontend_init,ERROR] FC7 Initialization Failed

18/03/2024 01:52

I tried to reprogram the device with 2GB of bar memory instead of 512KB. I don't think it was sucessful, but 'fe01' did see the device as a PCIe device

[root@fe01 pcimem]# lspci -vv | grep -A 40 "04:00.0"
pcilib: sysfs_read_vpd: read failed: Input/output error
04:00.0 Memory controller: Xilinx Corporation Device 7024
        Subsystem: Xilinx Corporation Device 0007
        Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Interrupt: pin A routed to IRQ 35
        Capabilities: [40] Power Management version 3
                Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1+,D2+,D3hot+,D3cold-)
                Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
        Capabilities: [48] MSI: Enable- Count=1/1 Maskable- 64bit+
                Address: 0000000000000000  Data: 0000
        Capabilities: [60] Express (v2) Endpoint, MSI 00
                DevCap: MaxPayload 512 bytes, PhantFunc 0, Latency L0s <64ns, L1 unlimited
                        ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset- SlotPowerLimit 225.000W
                DevCtl: Report errors: Correctable- Non-Fatal+ Fatal+ Unsupported-
                        RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+
                        MaxPayload 256 bytes, MaxReadReq 512 bytes
                DevSta: CorrErr- UncorrErr+ FatalErr- UnsuppReq+ AuxPwr- TransPend-
                LnkCap: Port #0, Speed 5GT/s, Width x4, ASPM L0s, Exit Latency L0s unlimited, L1 unlimited
                        ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp-
                LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk+
                        ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
                LnkSta: Speed 5GT/s, Width x4, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
                DevCap2: Completion Timeout: Range B, TimeoutDis-, LTR-, OBFF Not Supported
                DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
                LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-
                         Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
                         Compliance De-emphasis: -6dB
                LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1-
                         EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
        Capabilities: [100 v1] Device Serial Number 00-00-00-01-01-00-0a-35
        Kernel modules: xdma

[root@fe01 pcimem]# lspci -vv | grep -A 40 "04:00.0"
pcilib: sysfs_read_vpd: read failed: Input/output error
04:00.0 Memory controller: Xilinx Corporation Device 7024
        Subsystem: Xilinx Corporation Device 0007
        Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Interrupt: pin A routed to IRQ 35
        Capabilities: [40] Power Management version 3
                Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1+,D2+,D3hot+,D3cold-)
                Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
        Capabilities: [48] MSI: Enable- Count=1/1 Maskable- 64bit+
                Address: 0000000000000000  Data: 0000
        Capabilities: [60] Express (v2) Endpoint, MSI 00
                DevCap: MaxPayload 512 bytes, PhantFunc 0, Latency L0s <64ns, L1 unlimited
                        ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset- SlotPowerLimit 225.000W
                DevCtl: Report errors: Correctable- Non-Fatal+ Fatal+ Unsupported-
                        RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+
                        MaxPayload 256 bytes, MaxReadReq 512 bytes
                DevSta: CorrErr- UncorrErr+ FatalErr- UnsuppReq+ AuxPwr- TransPend-
                LnkCap: Port #0, Speed 5GT/s, Width x4, ASPM L0s, Exit Latency L0s unlimited, L1 unlimited
                        ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp-
                LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk+
                        ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
                LnkSta: Speed 5GT/s, Width x4, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
                DevCap2: Completion Timeout: Range B, TimeoutDis-, LTR-, OBFF Not Supported
                DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
                LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-
                         Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
                         Compliance De-emphasis: -6dB
                LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1-
                         EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
        Capabilities: [100 v1] Device Serial Number 00-00-00-01-01-00-0a-35
        Kernel modules: xdma

Notice there's no specified region (as opposed to below, where there's a 512K "Region 0"):

[root@fe01 pcimem]# lspci -vv | grep -A 34 "04:00.0"
04:00.0 Memory controller: Xilinx Corporation Device 7024
        Subsystem: Xilinx Corporation Device 0007
        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 0, Cache Line Size: 64 bytes
        Interrupt: pin A routed to IRQ 11
        Region 0: Memory at f5f80000 (32-bit, non-prefetchable) [size=512K]
        Capabilities: [40] Power Management version 3
                Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1+,D2+,D3hot+,D3cold-)
                Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
        Capabilities: [48] MSI: Enable- Count=1/1 Maskable- 64bit+
                Address: 0000000000000000  Data: 0000
        Capabilities: [60] Express (v2) Endpoint, MSI 00
                DevCap: MaxPayload 512 bytes, PhantFunc 0, Latency L0s <64ns, L1 unlimited
                        ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset- SlotPowerLimit 225.000W
                DevCtl: Report errors: Correctable- Non-Fatal+ Fatal+ Unsupported-
                        RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+
                        MaxPayload 256 bytes, MaxReadReq 512 bytes
                DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
                LnkCap: Port #0, Speed 5GT/s, Width x4, ASPM L0s, Exit Latency L0s unlimited, L1 unlimited
                        ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp-
                LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk+
                        ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
                LnkSta: Speed 5GT/s, Width x4, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
                DevCap2: Completion Timeout: Range B, TimeoutDis-, LTR-, OBFF Not Supported
                DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
                LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-
                         Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
                         Compliance De-emphasis: -6dB
                LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1-
                         EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
        Capabilities: [100 v1] Device Serial Number 00-00-00-01-01-00-0a-35

[root@fe01 pcimem]# lspci -vv | grep -A 34 "04:00.0"
04:00.0 Memory controller: Xilinx Corporation Device 7024
        Subsystem: Xilinx Corporation Device 0007
        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 0, Cache Line Size: 64 bytes
        Interrupt: pin A routed to IRQ 11
        Region 0: Memory at f5f80000 (32-bit, non-prefetchable) [size=512K]
        Capabilities: [40] Power Management version 3
                Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1+,D2+,D3hot+,D3cold-)
                Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
        Capabilities: [48] MSI: Enable- Count=1/1 Maskable- 64bit+
                Address: 0000000000000000  Data: 0000
        Capabilities: [60] Express (v2) Endpoint, MSI 00
                DevCap: MaxPayload 512 bytes, PhantFunc 0, Latency L0s <64ns, L1 unlimited
                        ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset- SlotPowerLimit 225.000W
                DevCtl: Report errors: Correctable- Non-Fatal+ Fatal+ Unsupported-
                        RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+
                        MaxPayload 256 bytes, MaxReadReq 512 bytes
                DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
                LnkCap: Port #0, Speed 5GT/s, Width x4, ASPM L0s, Exit Latency L0s unlimited, L1 unlimited
                        ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp-
                LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk+
                        ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
                LnkSta: Speed 5GT/s, Width x4, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
                DevCap2: Completion Timeout: Range B, TimeoutDis-, LTR-, OBFF Not Supported
                DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
                LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-
                         Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
                         Compliance De-emphasis: -6dB
                LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1-
                         EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
        Capabilities: [100 v1] Device Serial Number 00-00-00-01-01-00-0a-35

I tried to write and readback to an address, but was unable:

[root@fe01 pcimem]# sudo ./pcimem /dev/mem 4126670848 w 0xffffff12
/dev/mem opened.
Target offset is 0xf5f80000, page size is 4096
mmap(0, 4096, 0x3, 0x1, 3, 0xf5f80000)
PCI Memory mapped to address 0x7f758d6f4000.
Value at offset 0xF5F80000 (0x7f758d6f4000): 0xFFFFFFFF
Written 0xFFFFFF12; readback 0xFFFFFFFF

[root@fe01 pcimem]# sudo ./pcimem /dev/mem 4126670848 w 0xffffff12
/dev/mem opened.
Target offset is 0xf5f80000, page size is 4096
mmap(0, 4096, 0x3, 0x1, 3, 0xf5f80000)
PCI Memory mapped to address 0x7f758d6f4000.
Value at offset 0xF5F80000 (0x7f758d6f4000): 0xFFFFFFFF
Written 0xFFFFFF12; readback 0xFFFFFFFF

I'm not sure what I'm even doing here since I see no region to write to/read from.

18/03/2024 02:18

If I instead reprogram the device with 1MB of BAR memory, I can see the region

[root@fe01 ~]# lspci -vv | grep -A 40 "04:00.0"
pcilib: sysfs_read_vpd: read failed: Input/output error
04:00.0 Memory controller: Xilinx Corporation Device 7024
        Subsystem: Xilinx Corporation Device 0007
        Control: I/O+ Mem+ BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Interrupt: pin A routed to IRQ 35
        Region 0: Memory at f5f00000 (32-bit, non-prefetchable) [size=1M]
        Capabilities: [40] Power Management version 3
                Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1+,D2+,D3hot+,D3cold-)
                Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
        Capabilities: [48] MSI: Enable- Count=1/1 Maskable- 64bit+
                Address: 0000000000000000  Data: 0000
        Capabilities: [60] Express (v2) Endpoint, MSI 00
                DevCap: MaxPayload 512 bytes, PhantFunc 0, Latency L0s <64ns, L1 unlimited
                        ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset- SlotPowerLimit 225.000W
                DevCtl: Report errors: Correctable- Non-Fatal+ Fatal+ Unsupported-
                        RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+
                        MaxPayload 256 bytes, MaxReadReq 512 bytes
                DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
                LnkCap: Port #0, Speed 5GT/s, Width x4, ASPM L0s, Exit Latency L0s unlimited, L1 unlimited
                        ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp-
                LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk+
                        ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
                LnkSta: Speed 5GT/s, Width x4, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
                DevCap2: Completion Timeout: Range B, TimeoutDis-, LTR-, OBFF Not Supported
                DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
                LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-
                         Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
                         Compliance De-emphasis: -6dB
                LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1-
                         EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
        Capabilities: [100 v1] Device Serial Number 00-00-00-01-01-00-0a-35
        Kernel modules: xdma

[root@fe01 ~]# lspci -vv | grep -A 40 "04:00.0"
pcilib: sysfs_read_vpd: read failed: Input/output error
04:00.0 Memory controller: Xilinx Corporation Device 7024
        Subsystem: Xilinx Corporation Device 0007
        Control: I/O+ Mem+ BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Interrupt: pin A routed to IRQ 35
        Region 0: Memory at f5f00000 (32-bit, non-prefetchable) [size=1M]
        Capabilities: [40] Power Management version 3
                Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1+,D2+,D3hot+,D3cold-)
                Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
        Capabilities: [48] MSI: Enable- Count=1/1 Maskable- 64bit+
                Address: 0000000000000000  Data: 0000
        Capabilities: [60] Express (v2) Endpoint, MSI 00
                DevCap: MaxPayload 512 bytes, PhantFunc 0, Latency L0s <64ns, L1 unlimited
                        ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset- SlotPowerLimit 225.000W
                DevCtl: Report errors: Correctable- Non-Fatal+ Fatal+ Unsupported-
                        RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+
                        MaxPayload 256 bytes, MaxReadReq 512 bytes
                DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
                LnkCap: Port #0, Speed 5GT/s, Width x4, ASPM L0s, Exit Latency L0s unlimited, L1 unlimited
                        ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp-
                LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk+
                        ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
                LnkSta: Speed 5GT/s, Width x4, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
                DevCap2: Completion Timeout: Range B, TimeoutDis-, LTR-, OBFF Not Supported
                DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
                LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-
                         Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
                         Compliance De-emphasis: -6dB
                LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1-
                         EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
        Capabilities: [100 v1] Device Serial Number 00-00-00-01-01-00-0a-35
        Kernel modules: xdma

And I can write and readback in this case:

[root@fe01 pcimem]# sudo ./pcimem /dev/mem 4126146560 w 0xffffff12
/dev/mem opened.
Target offset is 0xf5f00000, page size is 4096
mmap(0, 4096, 0x3, 0x1, 3, 0xf5f00000)
PCI Memory mapped to address 0x7fd649f4b000.
Value at offset 0xF5F00000 (0x7fd649f4b000): 0x0
Written 0xFFFFFF12; readback 0xFFFFFF12
[root@fe01 pcimem]#

[root@fe01 pcimem]# sudo ./pcimem /dev/mem 4126146560 w 0xffffff12
/dev/mem opened.
Target offset is 0xf5f00000, page size is 4096
mmap(0, 4096, 0x3, 0x1, 3, 0xf5f00000)
PCI Memory mapped to address 0x7fd649f4b000.
Value at offset 0xF5F00000 (0x7fd649f4b000): 0x0
Written 0xFFFFFF12; readback 0xFFFFFF12
[root@fe01 pcimem]#

18/03/2024 02:31

AMC13Tool is throwing an error here:

  void AMC13::buildT1(const std::string& IP, const std::string& AD, 
              const int& IV, const bool& CH) {
    fpga_.resize(2);
    try {
      fpga_.at(T1) = new ipDev("hcal.crate42.T1", IP, AD, IV, CH);
    } catch(ipDev::exception& e) {
      printf("T1 %s %s %i %i\n", IP.c_str(), AD.c_str(), IV, CH);
      printf("Failed to build T1 ipDev object!\n");
      exit(1);
    }
  }

  void AMC13::buildT1(const std::string& IP, const std::string& AD, 
              const int& IV, const bool& CH) {
    fpga_.resize(2);
    try {
      fpga_.at(T1) = new ipDev("hcal.crate42.T1", IP, AD, IV, CH);
    } catch(ipDev::exception& e) {
      printf("T1 %s %s %i %i\n", IP.c_str(), AD.c_str(), IV, CH);
      printf("Failed to build T1 ipDev object!\n");
      exit(1);
    }
  }

Which means ipDev must fail at construction:

  ipDev::ipDev (const std::string& p_ID, const std::string& p_IP, const std::string& p_AddMap, 
        const int& IpBusV, const bool& ctrlHub) {
    // Build HwInterface ID
    ID = p_ID;
  
    // Build HwInterface URI
    if(!IpBusV) {
      if(ctrlHub)
    URI = "chtcp-1.3://localhost:10203?target="+p_IP+":50001";
      else
    URI = "ipbusudp-1.3://"+p_IP+":50001";
    }
    else if(IpBusV) {
      if(ctrlHub)
    URI = "chtcp-2.0://localhost:10203?target="+p_IP+":50001";
      else
    URI = "ipbusudp-2.0://"+p_IP+":50001";
    }
    
    // Build HwInterface Address Map
    ADD = "file://"+p_AddMap;
    // Construct HwInterface Object
    try {
      device = new uhal::HwInterface(uhal::ConnectionManager::getDevice(ID, URI, ADD));
    } catch(uhalException& e) {
        std::cerr << "Address map: " << ADD<< std::endl;
      std::cerr << "Caught Exception: " << e.what() << std::endl;
      throw exception(std::string("IPbus Creation Failure at URI "+URI));
    }
    // Handle Object's AddressTable entries
    createUhalNodeIDs();
  }

  ipDev::ipDev (const std::string& p_ID, const std::string& p_IP, const std::string& p_AddMap, 
        const int& IpBusV, const bool& ctrlHub) {
    // Build HwInterface ID
    ID = p_ID;
  
    // Build HwInterface URI
    if(!IpBusV) {
      if(ctrlHub)
    URI = "chtcp-1.3://localhost:10203?target="+p_IP+":50001";
      else
    URI = "ipbusudp-1.3://"+p_IP+":50001";
    }
    else if(IpBusV) {
      if(ctrlHub)
    URI = "chtcp-2.0://localhost:10203?target="+p_IP+":50001";
      else
    URI = "ipbusudp-2.0://"+p_IP+":50001";
    }
    
    // Build HwInterface Address Map
    ADD = "file://"+p_AddMap;
    // Construct HwInterface Object
    try {
      device = new uhal::HwInterface(uhal::ConnectionManager::getDevice(ID, URI, ADD));
    } catch(uhalException& e) {
        std::cerr << "Address map: " << ADD<< std::endl;
      std::cerr << "Caught Exception: " << e.what() << std::endl;
      throw exception(std::string("IPbus Creation Failure at URI "+URI));
    }
    // Handle Object's AddressTable entries
    createUhalNodeIDs();
  }

I added some print statements (as seen above) to see the exception has to do with the address table containing dots (there is a similar problem with using FC7.xml in the gm2daq, which makes me think at some point uHAL stopped liking dots in the address map file)

[root@dhcp-10-163-105-238 amc13StandaloneMAN_2014-05-12]#  bin/AMC13Tool -I 192.168.1.188
Connecting to AMC13...
T2 ip 192.168.1.188
T2 ip 192.168.1.188
T1 ip 192.168.1.189
Address map: file://map/AMC13_AddressTable_K7.xml
Caught Exception: Invalid node ID '1.0V_ANA_PWR' specified (contains dots)
T1 192.168.1.189 map/AMC13_AddressTable_K7.xml 1 1
Failed to build T1 ipDev object!

[root@dhcp-10-163-105-238 amc13StandaloneMAN_2014-05-12]#  bin/AMC13Tool -I 192.168.1.188
Connecting to AMC13...
T2 ip 192.168.1.188
T2 ip 192.168.1.188
T1 ip 192.168.1.189
Address map: file://map/AMC13_AddressTable_K7.xml
Caught Exception: Invalid node ID '1.0V_ANA_PWR' specified (contains dots)
T1 192.168.1.189 map/AMC13_AddressTable_K7.xml 1 1
Failed to build T1 ipDev object!

21/03/2024 00:16

Comparing our set-up with the CENPA DAQ, I think we have the Samtech cable plugged in correctly.
Our bank chip:

CENPA bank chip:

Our FC7 connection:

CENPA FC7 connection:

21/03/2024 00:50

Was able to set BARs size to 512MB

[root@fe01 ~]# lspci -vv | grep -A 40 "04:00.0"
pcilib: sysfs_read_vpd: read failed: Input/output error
04:00.0 Memory controller: Xilinx Corporation Device 7024
        Subsystem: Xilinx Corporation Device 0007
        Control: I/O+ Mem+ BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Interrupt: pin A routed to IRQ 35
        Region 0: Memory at c0000000 (32-bit, non-prefetchable) [size=512M]
        Capabilities: [40] Power Management version 3
                Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1+,D2+,D3hot+,D3cold-)
                Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
        Capabilities: [48] MSI: Enable- Count=1/1 Maskable- 64bit+
                Address: 0000000000000000  Data: 0000
        Capabilities: [60] Express (v2) Endpoint, MSI 00
                DevCap: MaxPayload 512 bytes, PhantFunc 0, Latency L0s <64ns, L1 unlimited
                        ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset- SlotPowerLimit 225.000W
                DevCtl: Report errors: Correctable- Non-Fatal+ Fatal+ Unsupported-
                        RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+
                        MaxPayload 256 bytes, MaxReadReq 512 bytes
                DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
                LnkCap: Port #0, Speed 5GT/s, Width x4, ASPM L0s, Exit Latency L0s unlimited, L1 unlimited
                        ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp-
                LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk+
                        ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
                LnkSta: Speed 5GT/s, Width x4, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
                DevCap2: Completion Timeout: Range B, TimeoutDis-, LTR-, OBFF Not Supported
                DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
                LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-
                         Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
                         Compliance De-emphasis: -6dB
                LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1-
                         EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
        Capabilities: [100 v1] Device Serial Number 00-00-00-01-01-00-0a-35
        Kernel modules: xdma

[root@fe01 ~]# lspci -vv | grep -A 40 "04:00.0"
pcilib: sysfs_read_vpd: read failed: Input/output error
04:00.0 Memory controller: Xilinx Corporation Device 7024
        Subsystem: Xilinx Corporation Device 0007
        Control: I/O+ Mem+ BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Interrupt: pin A routed to IRQ 35
        Region 0: Memory at c0000000 (32-bit, non-prefetchable) [size=512M]
        Capabilities: [40] Power Management version 3
                Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1+,D2+,D3hot+,D3cold-)
                Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
        Capabilities: [48] MSI: Enable- Count=1/1 Maskable- 64bit+
                Address: 0000000000000000  Data: 0000
        Capabilities: [60] Express (v2) Endpoint, MSI 00
                DevCap: MaxPayload 512 bytes, PhantFunc 0, Latency L0s <64ns, L1 unlimited
                        ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset- SlotPowerLimit 225.000W
                DevCtl: Report errors: Correctable- Non-Fatal+ Fatal+ Unsupported-
                        RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+
                        MaxPayload 256 bytes, MaxReadReq 512 bytes
                DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
                LnkCap: Port #0, Speed 5GT/s, Width x4, ASPM L0s, Exit Latency L0s unlimited, L1 unlimited
                        ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp-
                LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk+
                        ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
                LnkSta: Speed 5GT/s, Width x4, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
                DevCap2: Completion Timeout: Range B, TimeoutDis-, LTR-, OBFF Not Supported
                DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
                LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-
                         Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
                         Compliance De-emphasis: -6dB
                LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1-
                         EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
        Capabilities: [100 v1] Device Serial Number 00-00-00-01-01-00-0a-35
        Kernel modules: xdma

and write/readback:

[root@fe01 pcimem]# sudo ./pcimem /dev/mem 3221225472 w 0xffffff12
/dev/mem opened.
Target offset is 0xc0000000, page size is 4096
mmap(0, 4096, 0x3, 0x1, 3, 0xc0000000)
PCI Memory mapped to address 0x7fb23ee93000.
Value at offset 0xC0000000 (0x7fb23ee93000): 0x0
Written 0xFFFFFF12; readback 0xFFFFFF12
[root@fe01 pcimem]#

[root@fe01 pcimem]# sudo ./pcimem /dev/mem 3221225472 w 0xffffff12
/dev/mem opened.
Target offset is 0xc0000000, page size is 4096
mmap(0, 4096, 0x3, 0x1, 3, 0xc0000000)
PCI Memory mapped to address 0x7fb23ee93000.
Value at offset 0xC0000000 (0x7fb23ee93000): 0x0
Written 0xFFFFFF12; readback 0xFFFFFF12
[root@fe01 pcimem]#

21/03/2024 00:52

It looks like 1GB also works. I wonder what the issue with 2GB is:

[root@fe01 ~]# lspci -vv | grep -A 40 "04:00.0"
pcilib: sysfs_read_vpd: read failed: Input/output error
04:00.0 Memory controller: Xilinx Corporation Device 7024
        Subsystem: Xilinx Corporation Device 0007
        Control: I/O+ Mem+ BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Interrupt: pin A routed to IRQ 35
        Region 0: Memory at 80000000 (32-bit, non-prefetchable) [size=1G]
        Capabilities: [40] Power Management version 3
                Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1+,D2+,D3hot+,D3cold-)
                Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
        Capabilities: [48] MSI: Enable- Count=1/1 Maskable- 64bit+
                Address: 0000000000000000  Data: 0000
        Capabilities: [60] Express (v2) Endpoint, MSI 00
                DevCap: MaxPayload 512 bytes, PhantFunc 0, Latency L0s <64ns, L1 unlimited
                        ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset- SlotPowerLimit 225.000W
                DevCtl: Report errors: Correctable- Non-Fatal+ Fatal+ Unsupported-
                        RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+
                        MaxPayload 256 bytes, MaxReadReq 512 bytes
                DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
                LnkCap: Port #0, Speed 5GT/s, Width x4, ASPM L0s, Exit Latency L0s unlimited, L1 unlimited
                        ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp-
                LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk+
                        ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
                LnkSta: Speed 5GT/s, Width x4, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
                DevCap2: Completion Timeout: Range B, TimeoutDis-, LTR-, OBFF Not Supported
                DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
                LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-
                         Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
                         Compliance De-emphasis: -6dB
                LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1-
                         EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
        Capabilities: [100 v1] Device Serial Number 00-00-00-01-01-00-0a-35
        Kernel modules: xdma

[root@fe01 ~]# lspci -vv | grep -A 40 "04:00.0"
pcilib: sysfs_read_vpd: read failed: Input/output error
04:00.0 Memory controller: Xilinx Corporation Device 7024
        Subsystem: Xilinx Corporation Device 0007
        Control: I/O+ Mem+ BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Interrupt: pin A routed to IRQ 35
        Region 0: Memory at 80000000 (32-bit, non-prefetchable) [size=1G]
        Capabilities: [40] Power Management version 3
                Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1+,D2+,D3hot+,D3cold-)
                Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
        Capabilities: [48] MSI: Enable- Count=1/1 Maskable- 64bit+
                Address: 0000000000000000  Data: 0000
        Capabilities: [60] Express (v2) Endpoint, MSI 00
                DevCap: MaxPayload 512 bytes, PhantFunc 0, Latency L0s <64ns, L1 unlimited
                        ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset- SlotPowerLimit 225.000W
                DevCtl: Report errors: Correctable- Non-Fatal+ Fatal+ Unsupported-
                        RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+
                        MaxPayload 256 bytes, MaxReadReq 512 bytes
                DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
                LnkCap: Port #0, Speed 5GT/s, Width x4, ASPM L0s, Exit Latency L0s unlimited, L1 unlimited
                        ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp-
                LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk+
                        ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
                LnkSta: Speed 5GT/s, Width x4, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
                DevCap2: Completion Timeout: Range B, TimeoutDis-, LTR-, OBFF Not Supported
                DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
                LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-
                         Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
                         Compliance De-emphasis: -6dB
                LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1-
                         EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
        Capabilities: [100 v1] Device Serial Number 00-00-00-01-01-00-0a-35
        Kernel modules: xdma

and the write/readback:

[root@fe01 pcimem]# sudo ./pcimem /dev/mem 2147483648 w 0xffffff12
/dev/mem opened.
Target offset is 0x80000000, page size is 4096
mmap(0, 4096, 0x3, 0x1, 3, 0x80000000)
PCI Memory mapped to address 0x7f716ae30000.
Value at offset 0x80000000 (0x7f716ae30000): 0x0
Written 0xFFFFFF12; readback 0xFFFFFF12

[root@fe01 pcimem]# sudo ./pcimem /dev/mem 2147483648 w 0xffffff12
/dev/mem opened.
Target offset is 0x80000000, page size is 4096
mmap(0, 4096, 0x3, 0x1, 3, 0x80000000)
PCI Memory mapped to address 0x7f716ae30000.
Value at offset 0x80000000 (0x7f716ae30000): 0x0
Written 0xFFFFFF12; readback 0xFFFFFF12

21/03/2024 01:41

I was playing with talking with the board over PCIe. I got ChatGPT to make me a script that reads multiple words:

#include <stdio.h>
#include <stdlib.h>
#include <stdint.h>
#include <unistd.h>
#include <string.h>
#include <errno.h>
#include <signal.h>
#include <fcntl.h>
#include <ctype.h>
#include <termios.h>
#include <sys/types.h>
#include <sys/mman.h>
#include <sys/time.h>

#define PRINT_ERROR \
    do { \
        fprintf(stderr, "Error at line %d, file %s (%d) [%s]\n", \
        __LINE__, __FILE__, errno, strerror(errno)); exit(1); \
    } while(0)

#define MAP_SIZE_DEFAULT 4096UL
#define MAP_MASK (MAP_SIZE_DEFAULT - 1)

int main(int argc, char **argv) {
    int fd;
    void *map_base, *virt_addr;
    uint32_t read_result;
    char *filename;
    off_t target;
    size_t word_count = 0;
    size_t map_size = MAP_SIZE_DEFAULT; // Default map size
    struct timeval start, end;
    double time_taken;

    if(argc < 4 || argc > 5) {
        fprintf(stderr, "\nUsage:\t%s { sys file } { offset } { word count } [ map size ]\n"
            "\tsys file: sysfs file for the PCI resource to act on\n"
            "\toffset  : offset into PCI memory region to act upon\n"
            "\tword count: number of words to read\n"
            "\tmap size (optional): size of memory mapping in bytes (default: 4096)\n\n",
            argv[0]);
        exit(1);
    }
    
    filename = argv[1];
    target = strtoul(argv[2], 0, 0);
    word_count = atoi(argv[3]);

    if(argc == 5)
        map_size = atoi(argv[4]);

    if((fd = open(filename, O_RDWR | O_SYNC)) == -1) PRINT_ERROR;
    printf("%s opened.\n", filename);

    /* Map memory */
    map_base = mmap(0, map_size, PROT_READ, MAP_SHARED, fd, target & ~MAP_MASK);
    if(map_base == (void *) -1) PRINT_ERROR;
    printf("PCI Memory mapped to address 0x%08lx with size %lu bytes.\n", (unsigned long) map_base, map_size);

    virt_addr = map_base;

    printf("Reading %lu words from offset 0x%lx:\n", word_count, (unsigned long)target);

    // Measure time taken to read
    gettimeofday(&start, NULL);

    // Read and print the specified number of words
    for(size_t i = 0; i < word_count; ++i) {
        read_result = *((uint32_t *) virt_addr);
        printf("0x%08x ", read_result);
        virt_addr = (char *)virt_addr + sizeof(uint32_t); // Move to the next word
    }
    printf("\n");

    gettimeofday(&end, NULL);

    // Calculate time taken in milliseconds
    time_taken = ((end.tv_sec - start.tv_sec) * 1000 + (end.tv_usec - start.tv_usec) / 1000.0);

    printf("Read operation completed in %.2f milliseconds.\n", time_taken);

    if(munmap(map_base, map_size) == -1) PRINT_ERROR;
    close(fd);
    return 0;
}

#include <stdio.h>
#include <stdlib.h>
#include <stdint.h>
#include <unistd.h>
#include <string.h>
#include <errno.h>
#include <signal.h>
#include <fcntl.h>
#include <ctype.h>
#include <termios.h>
#include <sys/types.h>
#include <sys/mman.h>
#include <sys/time.h>

#define PRINT_ERROR \
    do { \
        fprintf(stderr, "Error at line %d, file %s (%d) [%s]\n", \
        __LINE__, __FILE__, errno, strerror(errno)); exit(1); \
    } while(0)

#define MAP_SIZE_DEFAULT 4096UL
#define MAP_MASK (MAP_SIZE_DEFAULT - 1)

int main(int argc, char **argv) {
    int fd;
    void *map_base, *virt_addr;
    uint32_t read_result;
    char *filename;
    off_t target;
    size_t word_count = 0;
    size_t map_size = MAP_SIZE_DEFAULT; // Default map size
    struct timeval start, end;
    double time_taken;

    if(argc < 4 || argc > 5) {
        fprintf(stderr, "\nUsage:\t%s { sys file } { offset } { word count } [ map size ]\n"
            "\tsys file: sysfs file for the PCI resource to act on\n"
            "\toffset  : offset into PCI memory region to act upon\n"
            "\tword count: number of words to read\n"
            "\tmap size (optional): size of memory mapping in bytes (default: 4096)\n\n",
            argv[0]);
        exit(1);
    }
    
    filename = argv[1];
    target = strtoul(argv[2], 0, 0);
    word_count = atoi(argv[3]);

    if(argc == 5)
        map_size = atoi(argv[4]);

    if((fd = open(filename, O_RDWR | O_SYNC)) == -1) PRINT_ERROR;
    printf("%s opened.\n", filename);

    /* Map memory */
    map_base = mmap(0, map_size, PROT_READ, MAP_SHARED, fd, target & ~MAP_MASK);
    if(map_base == (void *) -1) PRINT_ERROR;
    printf("PCI Memory mapped to address 0x%08lx with size %lu bytes.\n", (unsigned long) map_base, map_size);

    virt_addr = map_base;

    printf("Reading %lu words from offset 0x%lx:\n", word_count, (unsigned long)target);

    // Measure time taken to read
    gettimeofday(&start, NULL);

    // Read and print the specified number of words
    for(size_t i = 0; i < word_count; ++i) {
        read_result = *((uint32_t *) virt_addr);
        printf("0x%08x ", read_result);
        virt_addr = (char *)virt_addr + sizeof(uint32_t); // Move to the next word
    }
    printf("\n");

    gettimeofday(&end, NULL);

    // Calculate time taken in milliseconds
    time_taken = ((end.tv_sec - start.tv_sec) * 1000 + (end.tv_usec - start.tv_usec) / 1000.0);

    printf("Read operation completed in %.2f milliseconds.\n", time_taken);

    if(munmap(map_base, map_size) == -1) PRINT_ERROR;
    close(fd);
    return 0;
}

I'm pretty sure it works because I can write to registers with the pcimem script and see them appear when I read out, for example:
To write the first 4 byte word:

[root@fe01 pcimem]# sudo ./pcimem /dev/mem 2147483648 w 0xffffff12
/dev/mem opened.
Target offset is 0x80000000, page size is 4096
mmap(0, 4096, 0x3, 0x1, 3, 0x80000000)
PCI Memory mapped to address 0x7f0a17ba3000.
Value at offset 0x80000000 (0x7f0a17ba3000): 0xFFFFFF12
Written 0xFFFFFF12; readback 0xFFFFFF12
[root@fe01 pcimem]#

[root@fe01 pcimem]# sudo ./pcimem /dev/mem 2147483648 w 0xffffff12
/dev/mem opened.
Target offset is 0x80000000, page size is 4096
mmap(0, 4096, 0x3, 0x1, 3, 0x80000000)
PCI Memory mapped to address 0x7f0a17ba3000.
Value at offset 0x80000000 (0x7f0a17ba3000): 0xFFFFFF12
Written 0xFFFFFF12; readback 0xFFFFFF12
[root@fe01 pcimem]#

To write the second 4 byte word:

[root@fe01 pcimem]# sudo ./pcimem /dev/mem 2147483652 w 0xffffff13
/dev/mem opened.
Target offset is 0x80000004, page size is 4096
mmap(0, 4096, 0x3, 0x1, 3, 0x80000004)
PCI Memory mapped to address 0x7fc7f1ba1000.
Value at offset 0x80000004 (0x7fc7f1ba1004): 0xFFFFFF12
Written 0xFFFFFF13; readback 0xFFFFFF13

[root@fe01 pcimem]# sudo ./pcimem /dev/mem 2147483652 w 0xffffff13
/dev/mem opened.
Target offset is 0x80000004, page size is 4096
mmap(0, 4096, 0x3, 0x1, 3, 0x80000004)
PCI Memory mapped to address 0x7fc7f1ba1000.
Value at offset 0x80000004 (0x7fc7f1ba1004): 0xFFFFFF12
Written 0xFFFFFF13; readback 0xFFFFFF13

Now we can read the first 3 4-byte words:

[root@fe01 pcimem]# sudo ./pcie_read_dynamic /dev/mem 2147483648 3
/dev/mem opened.
PCI Memory mapped to address 0x7f1077abf000 with size 16384 bytes.
Reading 3 words from offset 0x80000000:
0xffffff12 0xffffff13 0x00000000
Read operation completed in 0.02 milliseconds.

[root@fe01 pcimem]# sudo ./pcie_read_dynamic /dev/mem 2147483648 3
/dev/mem opened.
PCI Memory mapped to address 0x7f1077abf000 with size 16384 bytes.
Reading 3 words from offset 0x80000000:
0xffffff12 0xffffff13 0x00000000
Read operation completed in 0.02 milliseconds.

I wrote one more word further out too and checked it:

[root@fe01 pcimem]# sudo ./pcimem /dev/mem 2147484052 w 0xffffff14
/dev/mem opened.
Target offset is 0x80000194, page size is 4096
mmap(0, 4096, 0x3, 0x1, 3, 0x80000194)
PCI Memory mapped to address 0x7fe1af562000.
Value at offset 0x80000194 (0x7fe1af562194): 0xFFFFFF12
Written 0xFFFFFF14; readback 0xFFFFFF14

[root@fe01 pcimem]# sudo ./pcimem /dev/mem 2147484052 w 0xffffff14
/dev/mem opened.
Target offset is 0x80000194, page size is 4096
mmap(0, 4096, 0x3, 0x1, 3, 0x80000194)
PCI Memory mapped to address 0x7fe1af562000.
Value at offset 0x80000194 (0x7fe1af562194): 0xFFFFFF12
Written 0xFFFFFF14; readback 0xFFFFFF14

[root@fe01 pcimem]# sudo ./pcie_read_dynamic /dev/mem 2147483648 110
/dev/mem opened.
PCI Memory mapped to address 0x7f2df0c91000 with size 4096 bytes.
Reading 110 words from offset 0x80000000:
0xffffff12 0xffffff13 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0xffffff14 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000
Read operation completed in 0.14 milliseconds.

[root@fe01 pcimem]# sudo ./pcie_read_dynamic /dev/mem 2147483648 110
/dev/mem opened.
PCI Memory mapped to address 0x7f2df0c91000 with size 4096 bytes.
Reading 110 words from offset 0x80000000:
0xffffff12 0xffffff13 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0xffffff14 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000
Read operation completed in 0.14 milliseconds.

However, I noticed that you can only read 512 4-byte words at a time before things start "repeating".There is a max payload size of 512 bytes (as seen in lscpi output further up), but this seems like a different affect?

[root@fe01 pcimem]# sudo ./pcie_read_dynamic /dev/mem 2147483648 516
/dev/mem opened.
PCI Memory mapped to address 0x7fadda667000 with size 4096 bytes.
Reading 516 words from offset 0x80000000:
0xffffff12 0xffffff13 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0xffffff14 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0xffffff12 0xffffff13 0x00000000 0x00000000
Read operation completed in 0.48 milliseconds.

[root@fe01 pcimem]# sudo ./pcie_read_dynamic /dev/mem 2147483648 516
/dev/mem opened.
PCI Memory mapped to address 0x7fadda667000 with size 4096 bytes.
Reading 516 words from offset 0x80000000:
0xffffff12 0xffffff13 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0xffffff14 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0xffffff12 0xffffff13 0x00000000 0x00000000
Read operation completed in 0.48 milliseconds.

21/03/2024 01:59

I noticed with above we're only seeing a data rate of about 1 MB/s. This is the timed part:

    // Measure time taken to read
    gettimeofday(&start, NULL);

    // Read and print the specified number of words
    for(size_t i = 0; i < word_count; ++i) {
        read_result = *((uint32_t *) virt_addr);
        printf("0x%08x ", read_result);
        virt_addr = (char *)virt_addr + sizeof(uint32_t); // Move to the next word
    }
    printf("\n");

    gettimeofday(&end, NULL);

    // Measure time taken to read
    gettimeofday(&start, NULL);

    // Read and print the specified number of words
    for(size_t i = 0; i < word_count; ++i) {
        read_result = *((uint32_t *) virt_addr);
        printf("0x%08x ", read_result);
        virt_addr = (char *)virt_addr + sizeof(uint32_t); // Move to the next word
    }
    printf("\n");

    gettimeofday(&end, NULL);

But I think this is actually just an artifact of printing. I editted the script to have more timing and removed the printout:

#include <stdio.h>
#include <stdlib.h>
#include <stdint.h>
#include <unistd.h>
#include <string.h>
#include <errno.h>
#include <signal.h>
#include <fcntl.h>
#include <ctype.h>
#include <termios.h>
#include <sys/types.h>
#include <sys/mman.h>
#include <sys/time.h>

#define PRINT_ERROR \
    do { \
        fprintf(stderr, "Error at line %d, file %s (%d) [%s]\n", \
        __LINE__, __FILE__, errno, strerror(errno)); exit(1); \
    } while(0)

#define MAP_SIZE_DEFAULT 4096UL
#define MAP_MASK (MAP_SIZE_DEFAULT - 1)

int main(int argc, char **argv) {
    int fd;
    void *map_base, *virt_addr;
    uint32_t *read_result;
    char *filename;
    off_t target;
    size_t word_count = 0;
    size_t map_size = MAP_SIZE_DEFAULT; // Default map size
    struct timeval start, end;
    long time_taken_microseconds;

    if(argc < 4 || argc > 5) {
        fprintf(stderr, "\nUsage:\t%s { sys file } { offset } { word count } [ map size ]\n"
            "\tsys file: sysfs file for the PCI resource to act on\n"
            "\toffset  : offset into PCI memory region to act upon\n"
            "\tword count: number of words to read\n"
            "\tmap size (optional): size of memory mapping in bytes (default: 4096)\n\n",
            argv[0]);
        exit(1);
    }
    
    filename = argv[1];
    target = strtoul(argv[2], 0, 0);
    word_count = atoi(argv[3]);

    if(argc == 5)
        map_size = atoi(argv[4]);

    gettimeofday(&start, NULL);
    if((fd = open(filename, O_RDWR | O_SYNC)) == -1) PRINT_ERROR;
    gettimeofday(&end, NULL);
    time_taken_microseconds = ((end.tv_sec - start.tv_sec) * 1000000 + (end.tv_usec - start.tv_usec));
    printf("%s opened in %ld microseconds.\n", filename, time_taken_microseconds);

    gettimeofday(&start, NULL);
    /* Map memory */
    map_base = mmap(0, map_size, PROT_READ, MAP_SHARED, fd, target & ~MAP_MASK);
    gettimeofday(&end, NULL);
    time_taken_microseconds = ((end.tv_sec - start.tv_sec) * 1000000 + (end.tv_usec - start.tv_usec));
    printf("PCI Memory mapped to address 0x%08lx with size %lu bytes in %ld microseconds.\n", (unsigned long) map_base, map_size, time_taken_microseconds);

    virt_addr = map_base;

    printf("Reading %lu words from offset 0x%lx in bulk mode:\n", word_count, (unsigned long)target);

    gettimeofday(&start, NULL);
    // Read data in bulk mode
    read_result = (uint32_t *)virt_addr;
    for(size_t i = 0; i < word_count; ++i) {
        //printf("0x%08x ", read_result[i]);
    }
    printf("\n");
    gettimeofday(&end, NULL);
    time_taken_microseconds = ((end.tv_sec - start.tv_sec) * 1000000 + (end.tv_usec - start.tv_usec));
    printf("Read operation completed in %ld microseconds.\n", time_taken_microseconds);

    gettimeofday(&start, NULL);
    if(munmap(map_base, map_size) == -1) PRINT_ERROR;
    gettimeofday(&end, NULL);
    time_taken_microseconds = ((end.tv_sec - start.tv_sec) * 1000000 + (end.tv_usec - start.tv_usec));
    printf("Memory unmapped in %ld microseconds.\n", time_taken_microseconds);

    gettimeofday(&start, NULL);
    close(fd);
    gettimeofday(&end, NULL);
    time_taken_microseconds = ((end.tv_sec - start.tv_sec) * 1000000 + (end.tv_usec - start.tv_usec));
    printf("%s closed in %ld microseconds.\n", filename, time_taken_microseconds);

    return 0;
}

#include <stdio.h>
#include <stdlib.h>
#include <stdint.h>
#include <unistd.h>
#include <string.h>
#include <errno.h>
#include <signal.h>
#include <fcntl.h>
#include <ctype.h>
#include <termios.h>
#include <sys/types.h>
#include <sys/mman.h>
#include <sys/time.h>

#define PRINT_ERROR \
    do { \
        fprintf(stderr, "Error at line %d, file %s (%d) [%s]\n", \
        __LINE__, __FILE__, errno, strerror(errno)); exit(1); \
    } while(0)

#define MAP_SIZE_DEFAULT 4096UL
#define MAP_MASK (MAP_SIZE_DEFAULT - 1)

int main(int argc, char **argv) {
    int fd;
    void *map_base, *virt_addr;
    uint32_t *read_result;
    char *filename;
    off_t target;
    size_t word_count = 0;
    size_t map_size = MAP_SIZE_DEFAULT; // Default map size
    struct timeval start, end;
    long time_taken_microseconds;

    if(argc < 4 || argc > 5) {
        fprintf(stderr, "\nUsage:\t%s { sys file } { offset } { word count } [ map size ]\n"
            "\tsys file: sysfs file for the PCI resource to act on\n"
            "\toffset  : offset into PCI memory region to act upon\n"
            "\tword count: number of words to read\n"
            "\tmap size (optional): size of memory mapping in bytes (default: 4096)\n\n",
            argv[0]);
        exit(1);
    }
    
    filename = argv[1];
    target = strtoul(argv[2], 0, 0);
    word_count = atoi(argv[3]);

    if(argc == 5)
        map_size = atoi(argv[4]);

    gettimeofday(&start, NULL);
    if((fd = open(filename, O_RDWR | O_SYNC)) == -1) PRINT_ERROR;
    gettimeofday(&end, NULL);
    time_taken_microseconds = ((end.tv_sec - start.tv_sec) * 1000000 + (end.tv_usec - start.tv_usec));
    printf("%s opened in %ld microseconds.\n", filename, time_taken_microseconds);

    gettimeofday(&start, NULL);
    /* Map memory */
    map_base = mmap(0, map_size, PROT_READ, MAP_SHARED, fd, target & ~MAP_MASK);
    gettimeofday(&end, NULL);
    time_taken_microseconds = ((end.tv_sec - start.tv_sec) * 1000000 + (end.tv_usec - start.tv_usec));
    printf("PCI Memory mapped to address 0x%08lx with size %lu bytes in %ld microseconds.\n", (unsigned long) map_base, map_size, time_taken_microseconds);

    virt_addr = map_base;

    printf("Reading %lu words from offset 0x%lx in bulk mode:\n", word_count, (unsigned long)target);

    gettimeofday(&start, NULL);
    // Read data in bulk mode
    read_result = (uint32_t *)virt_addr;
    for(size_t i = 0; i < word_count; ++i) {
        //printf("0x%08x ", read_result[i]);
    }
    printf("\n");
    gettimeofday(&end, NULL);
    time_taken_microseconds = ((end.tv_sec - start.tv_sec) * 1000000 + (end.tv_usec - start.tv_usec));
    printf("Read operation completed in %ld microseconds.\n", time_taken_microseconds);

    gettimeofday(&start, NULL);
    if(munmap(map_base, map_size) == -1) PRINT_ERROR;
    gettimeofday(&end, NULL);
    time_taken_microseconds = ((end.tv_sec - start.tv_sec) * 1000000 + (end.tv_usec - start.tv_usec));
    printf("Memory unmapped in %ld microseconds.\n", time_taken_microseconds);

    gettimeofday(&start, NULL);
    close(fd);
    gettimeofday(&end, NULL);
    time_taken_microseconds = ((end.tv_sec - start.tv_sec) * 1000000 + (end.tv_usec - start.tv_usec));
    printf("%s closed in %ld microseconds.\n", filename, time_taken_microseconds);

    return 0;
}

output:

[root@fe01 pcimem]# sudo ./pcie_read_dynamic /dev/mem 2147483648 512
/dev/mem opened in 6 microseconds.
PCI Memory mapped to address 0x7f2302115000 with size 4096 bytes in 13 microseconds.
Reading 512 words from offset 0x80000000 in bulk mode:

Read operation completed in 7 microseconds.
Memory unmapped in 7 microseconds.
/dev/mem closed in 3 microseconds.

[root@fe01 pcimem]# sudo ./pcie_read_dynamic /dev/mem 2147483648 512
/dev/mem opened in 6 microseconds.
PCI Memory mapped to address 0x7f2302115000 with size 4096 bytes in 13 microseconds.
Reading 512 words from offset 0x80000000 in bulk mode:

Read operation completed in 7 microseconds.
Memory unmapped in 7 microseconds.
/dev/mem closed in 3 microseconds.

Though, this doesn't seem right because it doesn't scale, see trying to read 1 word:

[root@fe01 pcimem]# sudo ./pcie_read_dynamic /dev/mem 2147483648 1
/dev/mem opened in 6 microseconds.
PCI Memory mapped to address 0x7f897ac92000 with size 4096 bytes in 13 microseconds.
Reading 1 words from offset 0x80000000 in bulk mode:

Read operation completed in 11 microseconds.
Memory unmapped in 7 microseconds.
/dev/mem closed in 4 microseconds.
[root@fe01 pcimem]#

[root@fe01 pcimem]# sudo ./pcie_read_dynamic /dev/mem 2147483648 1
/dev/mem opened in 6 microseconds.
PCI Memory mapped to address 0x7f897ac92000 with size 4096 bytes in 13 microseconds.
Reading 1 words from offset 0x80000000 in bulk mode:

Read operation completed in 11 microseconds.
Memory unmapped in 7 microseconds.
/dev/mem closed in 4 microseconds.
[root@fe01 pcimem]#

The size of the map doesn't seem to matter either (set to 4 bytes here):

[root@fe01 pcimem]# sudo ./pcie_read_dynamic /dev/mem 2147483648 4 4
/dev/mem opened in 7 microseconds.
PCI Memory mapped to address 0x7fcc26247000 with size 4 bytes in 13 microseconds.
Reading 4 words from offset 0x80000000 in bulk mode:

Read operation completed in 10 microseconds.
Memory unmapped in 7 microseconds.
/dev/mem closed in 4 microseconds.
[root@fe01 pcimem]#

[root@fe01 pcimem]# sudo ./pcie_read_dynamic /dev/mem 2147483648 4 4
/dev/mem opened in 7 microseconds.
PCI Memory mapped to address 0x7fcc26247000 with size 4 bytes in 13 microseconds.
Reading 4 words from offset 0x80000000 in bulk mode:

Read operation completed in 10 microseconds.
Memory unmapped in 7 microseconds.
/dev/mem closed in 4 microseconds.
[root@fe01 pcimem]#

I believe what's happening here is these timings are just overhead of these system operation of interacting with the memory map on the host computer. This script does not actually deal with any data transfer, rather, I think it incites data transfer (write to a virtual address mapped to target address, then some controller handles copying the data in the virtual address over). This makes timing the actual data transfer rate not as trivial as what I tried above.